System and method of generating a 3d representation of an object

ABSTRACT

The disclosed systems, structures, and methods are directed to generating a three-dimensional (3D) representation of an object, the method comprising accessing a 3D point cloud reconstruction of the object, accessing a 3D model template, the 3D model template defining a generic version of the object, aligning, in a geometrical space, the 3D model template with respect to the 3D point cloud reconstruction, adjusting a scale of the 3D model template, and applying local deformations to a surface of the 3D model template.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.62/952,193, entitled “SYSTEM AND METHOD OF GENERATING A 3DREPRESENTATION OF AN OBJECT”, filed on Dec. 20, 2019, and also claimspriority to U.S. Provisional Patent Application Ser. No. 62/855,451,filed on May 31, 2019 and entitled “SYSTEM AND METHOD OF GENERATING A 3DREPRESENTATION OF AN OBJECT”, the contents of both of which areincorporated herein by reference in jurisdictions allowing suchincorporation.

FIELD

The present technology relates to systems and methods of generating a 3Drepresentation of an object.

BACKGROUND

Three-dimensional (“3D”) digital data may be produced by a variety ofdevices that involve three-dimensional scanning or sampling and/ornumerical modeling. In one example, 3D laser scanners generate 3Ddigital data. A long range laser scanner is fixed in one location androtated to scan objects around it. Alternatively, a short-range laserscanner is mounted on a device that moves around an object whilescanning it. In any of the scenarios, the location of each point scannedis represented as a polar coordinate since the angle between the scannerand the object and distance from the scanner to the object are known.The polar coordinates are then converted to 3D Cartesian coordinates andstored along with a corresponding intensity or color value for the datapoint collected by the scanner.

Other examples to generate 3D digital data are depth cameras or 3Dscanner to generate 3D digital data by collecting a complete point setof (x, y, z) locations that represent the shape of an object. Oncecollected, these point sets, also known as 3D point clouds, are sent toan image rendering system, which then processes the point data togenerate a 3D representation of the object.

Typical systems and methods to capture 3D point clouds and then generate3D representation of the object require specialized, cumbersome andcostly hardware equipment. To this end, there is an interest indeveloping cost effective systems and methods to generate 3Drepresentation from 3D point clouds.

SUMMARY

Embodiments of the present technology have been developed based ondevelopers' appreciation of at least one technical problem associatedwith the prior art solutions.

For example, even though the prior art suggest techniques to generate 3Dpoint clouds and then from the 3D point clouds generating a 3Drepresentation of the object. However, such techniques requirespecialized, cumbersome and costly hardware equipment such as 3D laserscanners. Moreover, efficient generation of 3D representation of theobject from the 3D point clouds is still a pending concern.

In developing the present technology, developers noted thatnon-specialized hardware, such as a mobile device comprising a camera(e.g., an iPhone® from Apple or a Galaxy® from Samsung) may be used toconduct an acquisition of a 3D point cloud reconstruction. The 3D pointcloud reconstruction may then be used in combination with a 3D modeltemplate to generate a 3D representation of the object.

In accordance with a first broad aspect of the present technology, thereis provided a method of generating a three-dimensional (3D)representation of an object, the method comprising accessing a 3D pointcloud reconstruction of the object, accessing a 3D model template, the3D model template defining a generic version of the object, aligning, ina geometrical space, the 3D model template with respect to the 3D pointcloud reconstruction, adjusting a scale of the 3D model template, andapplying local deformations to a surface of the 3D model template.

In some embodiments of the method, wherein accessing the 3D point cloudreconstruction of the object comprises applying a denoising routine.

In some embodiments of the method, wherein accessing the 3D modeltemplate is performed in accordance with a selection routine.

In some embodiments of the method, wherein the 3D model template is a 3Dmesh model.

In some embodiments of the method, wherein the geometrical space isassociated with a cartesian coordinate system and/or a cylindricalcoordinate system and/or a spherical coordinate system.

In some embodiments of the method, wherein aligning the 3D modeltemplate with respect to the 3D point cloud reconstruction comprisesapplying a rotation and/or a translation.

In some embodiments of the method, wherein aligning the 3D modeltemplate with respect to the 3D point cloud reconstruction comprises (i)aligning, in translation, a first center of gravity associated with the3D model template and a second center of gravity associated with the 3Dpoint cloud reconstruction, and (ii) aligning, in rotation, the 3D modeltemplate with respect to the 3D point cloud reconstruction is based onat least one of a minimum distance between the 3D model template and the3D point cloud reconstruction, Principal Components Analysis, and/or 2Dprojections alignments.

In some embodiments of the method, wherein aligning, in rotation, the 3Dmodel template with respect to the 3D point cloud reconstruction basedon Principal Components Analysis comprises identifying 3 main axis ofthe 3D model template to align the 3 main axis with the axis of the 3Dpoint cloud reconstruction.

In some embodiments of the method, wherein aligning, in rotation, the 3Dmodel template with respect to the 3D point cloud reconstruction basedon 2D projections alignments comprises computing 2D pictures from the 3Dmodel template in a combination of different points of view and matchthe 2D pictures to match the points of views.

In some embodiments of the method, wherein adjusting the scale of the 3Dmodel comprises executing an iterative closest point (ICP) algorithm.

In some embodiments of the method, wherein adjusting the scale of the 3Dmodel is performed iteratively and comprises one of (1) modifying ascale of the 3D model template, (2) applying/refining a rotation to the3D model template and/or (3) applying/refining a translation to the 3Dmodel template, calculating a distance between the 3D model template andthe 3D point cloud reconstruction, and wherein the iterating stops basedon a calculated minimum distance.

In some embodiments of the method, applying local deformations to thesurface of the 3D model template comprises determining, for at leastsome of the points of the 3D model template, normal axis extending fromeach of the at least some of the points of the 3D model template,displacing the at least some of the points of the 3D model templatealong corresponding normal axis of a computed amplitude, the displacingfurther comprising displacing neighboring points located within a radiusof the at least some of the points of the 3D model template.

In some embodiments of the method, wherein the computed amplitude isbased on a scalar product of a normalized normal vector to the at leastsome of the points of the 3D model template and a vector linking a pointto be displaced in the 3D model template and a centroid in the 3D pointcloud reconstruction.

In some embodiments, the method further comprises generating at leasttwo 3D representations of at least two objects and combining the atleast two 3D representations.

In some embodiments, the method further comprises identifying lowquality deformations, selecting parts to be corrected, correcting theparts.

In some embodiments of the method, wherein low quality deformations areidentified by operating a quality function.

In some embodiments of the method, wherein selecting parts to becorrected is based on distances distribution and/or angles distributionbetween the 3D model template and a deformed 3D model template.

In some embodiments of the method, wherein applying the localdeformations to the surface of the 3D model template comprises computinga quality level of the local deformations, and correcting the localdeformations based on the computed quality level.

In some embodiments of the method, wherein the correcting the localdeformations based on the computed quality level further comprisescorrecting the local deformations if a computed quality level is below athreshold.

In accordance with a second broad aspect of the present technology,there is provided a system for generating a three-dimensional (3D)representation of an object, the system comprising a processor, anon-transitory computer-readable medium comprising instructions, theprocessor, upon executing the instructions, being configured to causethe processor to access a 3D point cloud reconstruction of the object,access a 3D model template, the 3D model template defining a genericversion of the object, align, in a geometrical space, the 3D modeltemplate with respect to the 3D point cloud reconstruction, adjust ascale of the 3D model template, and apply local deformations to asurface of the 3D model template.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects and advantages of the presenttechnology will become better understood with regard to the followingdescription, appended claims and accompanying drawings where:

FIG. 1 is an illustration of an exemplary environment for executing amethod of generating a 3D representation of an object;

FIG. 2 is an illustration of an exemplary embodiment of a 3Drepresentation of an object;

FIGS. 3 and 4 are illustrations of exemplary embodiments of 3D pointcloud reconstructions;

FIG. 5 is an illustration of an exemplary embodiment of a 3D modeltemplate;

FIG. 6 is an illustration of a method carried out in accordance withnon-limiting embodiments of the present technology;

FIG. 7 illustrates examples of initial alignments in accordance withembodiments of the present technology;

FIG. 8 illustrates examples of an alternative alignment routine inaccordance with embodiments of the present technology;

FIG. 9 illustrates an example of adjusting a scale of the 3D modeltemplate in accordance with embodiments of the present technology;

FIG. 10 illustrates evolution of a calculated minimum distance withrespect to various scale values in accordance with embodiments of thepresent technology;

FIG. 11 illustrates two examples of 3D point cloud reconstructions inaccordance with embodiments of the present technology;

FIG. 12 illustrates vertices from a 3D model template and points from a3D point cloud reconstruction in accordance with embodiments of thepresent technology;

FIG. 13 illustrates a “bell” function and scenarios of displacements ofvertices in accordance with embodiments of the present technology;

FIG. 14 illustrates a first 3D model template before and after morphingin accordance with embodiments of the present technology;

FIG. 15 illustrates a second 3D model template before and after morphingin accordance with embodiments of the present technology;

FIG. 16 illustrates another example of context for applying the presenttechnology;

FIG. 17 illustrates a 3D point cloud reconstruction and a 3Drepresentation generated from the 3D point cloud reconstruction inaccordance with embodiments of the present technology;

FIG. 18 illustrates quality measures for identifying parts to becorrected in accordance with embodiments of the present technology;

FIG. 19 illustrates an embodiment of steps for identifying parts to becorrected and parts after correction in accordance with embodiments ofthe present technology; and

FIG. 20 is an illustration of additional steps of a method carried outin accordance with non-limiting embodiments of the present technology.

DETAILED DESCRIPTION OF THE DRAWINGS

Various exemplary embodiments of the described technology will bedescribed more fully hereinafter with reference to the accompanyingdrawings, in which exemplary embodiments are shown. The presentinventive concept may, however, be embodied in many different forms andshould not be construed as limited to the exemplary embodiments setforth herein. Rather, these exemplary embodiments are provided so thatthe disclosure will be thorough and complete, and will fully convey thescope of the present inventive concept to those skilled in the art. Inthe drawings, the sizes and relative sizes of layers and regions may beexaggerated for clarity. Like numerals refer to like elementsthroughout.

It will be understood that, although the terms first, second, third etc.may be used herein to describe various elements, these elements shouldnot be limited by these terms. These terms are used to distinguish oneelement from another. Thus, a first element discussed below could betermed a second element without departing from the teachings of thepresent inventive concept. As used herein, the term “and/or” includesany and all combinations of one or more of the associated listed items.

It will be understood that when an element is referred to as being“connected” or “coupled” to another element, it can be directlyconnected or coupled to the other element or intervening elements may bepresent. In contrast, when an element is referred to as being “directlyconnected” or “directly coupled” to another element, there are nointervening elements present. Other words used to describe therelationship between elements should be interpreted in a like fashion(e.g., “between” versus “directly between,” “adjacent” versus “directlyadjacent,” etc.).

The terminology used herein is only intended to describe particularexemplary embodiments and is not intended to be limiting of the presentinventive concept. As used herein, the singular forms “a,” “an” and“the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. It will be further understood thatthe terms “comprises” and/or “comprising,” when used in thisspecification, specify the presence of stated features, integers, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof.

Moreover, all statements herein reciting principles, aspects, andimplementations of the present technology, as well as specific examplesthereof, are intended to encompass both structural and functionalequivalents thereof, whether they are currently known or developed inthe future. Thus, for example, it will be appreciated by those skilledin the art that any block diagrams herein represent conceptual views ofillustrative circuitry embodying the principles of the presenttechnology. Similarly, it will be appreciated that any flowcharts, flowdiagrams, state transition diagrams, pseudo-code, and the like representvarious processes which may be substantially represented incomputer-readable media and so executed by a computer or processor,whether or not such computer or processor is explicitly shown.

The functions of the various elements shown in the figures, includingany functional block labeled as a “processor”, may be provided throughthe use of dedicated hardware as well as hardware capable of executingsoftware in association with appropriate software. When provided by aprocessor, the functions may be provided by a single dedicatedprocessor, by a single shared processor, or by a plurality of individualprocessors, some of which may be shared. In some embodiments of thepresent technology, the processor may be a general purpose processor,such as a central processing unit (CPU) or a processor dedicated to aspecific purpose, such as a digital signal processor (DSP). Moreover,explicit use of the term a “processor” should not be construed to referexclusively to hardware capable of executing software, and mayimplicitly include, without limitation, application specific integratedcircuit (ASIC), field programmable gate array (FPGA), read-only memory(ROM) for storing software, random access memory (RAM), and non-volatilestorage. Other hardware, conventional and/or custom, may also beincluded.

Software modules, or simply modules which are implied to be software,may be represented herein as any combination of flowchart elements orother elements indicating performance of process steps and/or textualdescription. Such modules may be executed by hardware that is expresslyor implicitly shown. Moreover, it should be understood that module mayinclude for example, but without being limitative, computer programlogic, computer program instructions, software, stack, firmware,hardware circuitry or a combination thereof which provides the requiredcapabilities.

In the context of the present technology, a “3D representation of anobject” may refer to a digital 3D model of this object under the form ofa digital file or a digital group of files that describe the 3D geometryof an object. The description of an object's 3D geometry is given interms of a collection of points in a 3D space (hereafter called“vertices”, singular: “vertex”), those vertices being connected or notby various geometry entities such as triangles lines, planar or curvedsurface, etc. 3D models can be created by hand, algorithmically orscanned. Other information related to the 3D representation of the modelsuch as colors, textures, materials, shaders, etc. can be stored incomplementary files that will be associated to the main file describingthe geometry of the object. This is why a 3D model can be made of a“group of files”. Typically, a wooden aspect will be given by a separatepicture of a wood texture that will be linked to the geometry. In thepresent disclosure, we will use both scanned and handcrafted 3D models.The format of those digital files are numerous and the invention iscompatible to the most common ones. Examples of 3D representations ofobjects are illustrated at FIG. 2 in which 3D representations of humanfeet 202 and 204 are depicted.

In the context of the present technology, a “3D point cloudreconstruction” may refer to a simple 3D representation of an objectwere the vertices are not necessarily connected to each other. If theyare not connected to each other, the only information contained in thiskind of representation is the coordinates (e.g., x, y, z in the case ofa cartesian coordinate system) of each vertex, and its color (e.g., r,g, b). The 3D point cloud reconstruction is often used as the result ofa 3D scanning and a very common format for those files is the PolygonFile Format (PLY). 3D point cloud reconstructions are typically simple3D representations which are usually rough and barely used directly bymost users as they typically do not allow a realistic representation ofthe object but rather a set of 3D points without any relations with eachother besides their position and color. A first example of a 3D pointcloud reconstruction is illustrated at FIG. 3 in which a 3D point cloudreconstruction of a human foot 302 is depicted. A second example of a 3Dpoint cloud reconstruction is illustrated at FIG. 4 in which a 3D pointcloud reconstruction of a human foot 402 is depicted.

In the context of the present technology, a “3D model template” mayrefer to a complex 3D representation of an object were vertices areconnected together to form faces that will build the whole and completesurface of a 3D object. It is thus typically realistic and customizable.The present disclosure refers to a “3D model template” to betterillustrate aspects of the present technology allowing to modify acomplex 3D representation of a model to match simple 3D representation(3D “point clouds”) of this same object. Indeed, point clouds are easierto make from a connection to a real object (through a scanner or a setof pictures) than complex 3D models. Aspects of the present technologyrelates to deforming a complex 3D model (3D morphing) with all itsparameters in such a way that it fits to reality (point clouds).Reference is made to “template” as, in some embodiments of the presenttechnology, the “3D model template” defines a generic version of anobject or of a class of objects. As non-limiting examples, 3D modeltemplates may be a generic human foot, a generic body part, a genericbody, a generic part or component of machinery. An example of a 3D modeltemplate is illustrated at FIG. 5 in which a 3D model template of ahuman foot 502 is depicted.

The present disclosure illustrates embodiments of the technologyfocusing on the 3D reconstruction of a human foot for the purpose ofefficiently and easily generate 3D reconstructions without requiringspecialized equipment such as dedicated medical scanners. As a result,non-specialized hardware, such as a mobile device comprising a camera(e.g., an iPhone® from Apple or a Galaxy® from Samsung) may be used toconduct an acquisition of a 3D point cloud reconstruction. The 3D pointcloud reconstruction may then be used in combination with a 3D modeltemplate to generate a 3D representation in accordance with the presenttechnology.

The present technology may therefore provide a cost-efficient approachto the problem of generating accurate 3D representation of object whichwould have otherwise required specialized, cumbersome and costlyhardware equipment. The present technology may for example be suitablefor the generation of accurate 3D representation of human feet which maythen be used for the manufacturing of customized medical devices, suchas, but without being limitative, orthoses.

Even though the present technology is presented in the context ofgenerating highly reliable 3D representations of human feet from 3Dpoint cloud reconstructions and 3D models, it should be understood thatthis aspect is not limitative. To the contrary, the present technologymay be applied to the generation of 3D representations of any objects.As a non-limitative example, the present technology may be applied tothe generation of 3D representations of a complete body or of one ormore body parts. Such 3D representations of a body or of one or morebody parts may then be used for different applications, such as, butwithout being limitative, an assessment of physical conditions,recommendations for the purpose of physical training and/or for thepurpose of suggesting a diet based on the physical conditionsextrapolated from the 3D representations of the one or more body parts.

As another non-limitative example, the present technology may be appliedto the generation of 3D representations of mechanical parts frompictures or video sequences, for example in the context of generating 3Drepresentations of parts and/or components of machinery (e.g., parts ofan aircraft, parts of an engine, etc.). Such 3D representations, oncegenerated, may be used to populate and/or complete a database of 3Dobjects, such as a computer aided design (CAD) database. Othervariations of objects or applications may also be envisioned withoutdeparting from the scope of the present technology.

With these fundamentals in place, we will now consider some non-limitingexamples to illustrate various implementations of aspects of the presenttechnology.

FIG. 1 illustrates a diagram of a computing environment 100 inaccordance with an embodiment of the present technology is shown. Insome embodiments, the computing environment 100 may be implemented byany of a conventional personal computer, a computer dedicated tooperating generation of 3D representation of objects, a remote serverand/or an electronic device (such as, but not limited to, a mobiledevice, a tablet device, a server, a controller unit, a control device,a monitoring device etc.) and/or any combination thereof appropriate tothe relevant task at hand. In some embodiments, the computingenvironment 100 comprises various hardware components including one ormore single or multi-core processors collectively represented by aprocessor 110, a solid-state drive 120, a random access memory 130, acamera 132 and an input/output interface 150. The computing environment100 may be a computer specifically designed for operating generation of3D representation of objects. In some alternative embodiments, thecomputing environment 100 may be a generic computer system.

In some embodiments, the computing environment 100 may also be asub-system of one of the above-listed systems. In some otherembodiments, the computing environment 100 may be an “off the shelf”generic computer system. In some embodiments, the computing environment100 may also be distributed amongst multiple systems. In someembodiments, the computing environment 100 is virtualized in the “cloud”so that processing power and/or memory capacity may be scaled up or downdepending on actual needs for executing implementations of the presenttechnology. The computing environment 100 may also be specificallydedicated to the implementation of the present technology. As a personin the art of the present technology may appreciate, multiple variationsas to how the computing environment 100 is implemented may be envisionedwithout departing from the scope of the present technology.

Communication between the various components of the computingenvironment 100 may be enabled by one or more internal and/or externalbuses 160 (e.g. a PCI bus, universal serial bus, IEEE 1394 “Firewire”bus, SCSI bus, Serial-ATA bus, ARINC bus, etc.), to which the varioushardware components are electronically coupled.

The input/output interface 150 may allow enabling networkingcapabilities such as wire or wireless access. As an example, theinput/output interface 150 may comprise a networking interface such as,but not limited to, a network port, a network socket, a networkinterface controller and the like. Multiple examples of how thenetworking interface may be implemented will become apparent to theperson skilled in the art of the present technology. For example, butwithout being limitative, the networking interface may implementspecific physical layer and data link layer standard such as Ethernet,Fibre Channel, Wi-Fi or Token Ring. The specific physical layer and thedata link layer may provide a base for a full network, allowingcommunication among small groups of computers on the same local areanetwork (LAN) and large-scale network communications through routableprotocols, such as Internet Protocol (IP).

According to implementations of the present technology, the solid-statedrive 120 stores program instructions suitable for being loaded into therandom access memory 130 and executed by the processor 110 for executinggeneration of 3D representation of objects. For example, the programinstructions may be part of a library or an application.

Referring now concurrently to FIG. 6 and FIG. 7-15, some non-limitingexample instances of systems and computer-implemented methods used inconnection with generating a 3D representation of an object shall now bedescribed. More specifically, FIG. 6 shows flowcharts illustrating acomputer-implemented method 600 implementing embodiments of the presenttechnology. The computer-implemented method of FIG. 6 may comprise acomputer-implemented method executable by a processor of a computingenvironment, such as the computing environment 100 of FIG. 1, the methodcomprising a series of steps to be carried out by the computingenvironment.

The computer-implemented method of FIG. 6 may be carried out, forexample, by a processor executing program instructions having beenloaded, for example, into random access memories.

The method 600 starts at step 602 by accessing a 3D point cloudreconstruction of the object. As it may be appreciated, the 3D pointcloud reconstruction of the object may be accessed from a memory storingone or more 3D point cloud reconstructions of objects previouslyacquired through various methodologies which will become apparent to theperson skilled in the art of the present technology. In someembodiments, the 3D point cloud reconstruction of the object may havebeen generated from images or a video sequence depicting the object andcaptured through the camera of the device on which the method 600 isexecuted (e.g., the camera 132 of the device 100). The images or thevideo sequence may be acquired by a camera embedded in a mobile device.In such instances, the images or the video sequence may be subjected toone or more post-processing methods which allow generating point cloudsfrom raw images or video sequences. In some embodiments, the 3D pointcloud reconstruction may be subjected to a denoising routine allowing toincrease an accuracy of the points defining the 3D point cloudreconstruction and/or reduce a number of points which do not relate tothe object itself. The object represented in the context of the presentdisclosure is a human foot. As previously mentioned, this aspect is notlimitative, the present technology may equally be applicable to otherobjects (e.g., body parts, body, parts or components of machinery, etc).

At a step 604, the method 600 proceeds to accessing a 3D model templatewhich defines a generic version of the object. In some embodiments, the3D model template may accessed from a database of 3D models. A selectionroutine may be applied to select the one or more 3D models which aredeemed to be the most suitable for the particular context in which the3D reconstruction is to be generated. In the illustrated example, the 3Dmodel is a high resolution 3D mesh of a human foot, for example the 3Dmodel 502 of FIG. 5. In some embodiments, the selection routine maydetermine the context based on an analysis of the 3D point cloudreconstruction accessed at step 602. Other approaches may equally beenvisioned.

In some embodiments, the 3D model template which is a 3D mesh model maybe used as a 3D point cloud model. In some embodiments, the 3D pointcloud model may be created from the 3D mesh model.

At a step 606, the method 600 proceeds to aligning, in a geometricalspace, the 3D model template with respect to the 3D point cloudreconstruction. In some embodiments, the geometrical space is associatedwith a cartesian coordinate system and/or a cylindrical coordinatesystem and/or a spherical coordinate system. In some embodiments,aligning the 3D model template with respect to the 3D point cloudreconstruction comprises applying a translation and/or a rotation. Thetranslation may comprise aligning, in translation, a first center ofgravity associated with the 3D model template and a second center ofgravity associated with the 3D point cloud reconstruction. The rotationmay comprise aligning, in rotation, the 3D model template with respectto the 3D point cloud reconstruction based on a minimum distance betweenthe 3D model template and the 3D point cloud reconstruction. In someembodiments, the rotation alignment may be a combination of severalapproaches applied separately or combined together until a best initialrotation alignment is performed.

A first example of such alignment approach may be a method based onprincipal component analysis (PCA). Such method may allow identifyingamong other parameters 3 main axis of the 3D model so that the 3 mainaxis may be aligned with the axis of the 3D point cloud reconstructionusing, for example, the method of minimum distances. Each distance iscomputed for each combination of axis alignments and the lowest distancemay lead to the correct alignment. FIG. 7 illustrates a first example ofa wrong initial alignment 710 wherein a 3D model 712 is not properlyaligned with a 3D point cloud reconstruction 714. FIG. 7 alsoillustrates a second example of a proper initial alignment 720 wherein a3D model 722 is properly aligned with a 3D point cloud reconstruction724.

A second example of such alignment approach may be a method based on 2Dmasks projections alignment. Such method may consist in computing 2Dpictures from the 3D model in a combination of different points of viewand to match the pictures to match the points of views. It allowsdetermining the alignment of the 3 axis of the 3D model with the 3 axisof the 3D point cloud reconstruction. FIG. 8 illustrates an examplewherein images 810 and 812 are view from a similar view point andwherein images 812 and 814 are viewed from a different view point (andtherefore do not provide a match).

At a step 608, the method 600 proceeds to adjusting a scale of the 3Dmodel template. In some embodiments, the adjusting of the 3D modelcomprises executing an iterative closest point (ICP) algorithm. In someembodiments, the adjusting comprises iteratively executing sub-steps 610and 612. In some embodiments, sub-step 610 comprises modifying a scaleof the 3D model template, applying/refining a rotation to the 3D modeltemplate and/or applying/refining a translation to the 3D modeltemplate. In some embodiments, sub-step 612 comprises calculating adistance between the 3D model template and the 3D point cloudreconstruction. In some embodiments, the calculated distance may be oneof the closest point and the point to normal distance.

In some embodiments, determination that iteration of sub-steps 610 and612 is to be stopped (and that the best initial alignment is found) isbased on the calculated minimum distance at a step 614.

FIG. 9 illustrates an example of adjusting a scale of the 3D modeltemplate in accordance with step 608. In the illustrated example, afirst representation 910 illustrates a 3D model 912 associated with afirst scale and aligned with a 3D point cloud reconstruction 914. Asecond representation 920 illustrates a 3D model 922 associated with asecond scale and aligned with a 3D point cloud reconstruction 924. Ascan be seen in the second representation 920, the 3D model 922 is overscaled, so as to ensure that an appropriate (or best) scale value isfound. FIG. 10 illustrates evolution of the calculated minimum distancewith respect to various scale values. In the illustration of FIG. 10,the appropriate scale value is determined to be the one associated withthe minimum distance between the 3D model and the 3D point cloudreconstruction (i.e., a point 1002 on the curve).

It should be noted that besides having a metric reference in the scenebeing shot, another way to calculate an absolute metric may be to use agyroscope and an accelerometer if the device 210 is equipped with those.With combined data provided by accelerometers and/or gyroscopes nowembedded in most portable devices, it is possible, without any measureinformation in the 2D pictures, to calculate with sufficient accuracy,the size and measures (2D and/or 3D) of the 3D reconstructed objects.Absolute 2D dimensions may be provided with a precision of less than 5%of error, which is usable from the user's point of view. 3D volumes maybe computed with only one 2D measurement information (length, height,depth and/or width . . . ).

At a step 616, the method 600 proceeds to applying local deformations toa surface of the 3D model template. In some embodiments, the step 616comprises sub-steps 618 and 620. The step 618 comprises determining, forat least some of the points of the 3D model template, normal axisextending from each of the at least some of the points of the 3D modeltemplate. FIG. 11 illustrates two examples of 3D point cloudreconstructions 1102 and 1104. The 3D point cloud reconstruction 1104 isillustrated with normals projecting from points.

The step 620 comprises displacing at least some of the points (alsoreferred to as “vertex” or “vertices”) of the 3D model template alongcorresponding normal axis of a computed amplitude. In some embodiments,the displacing further comprises displacing neighboring points locatedwithin a radius of the at least some of the points of the 3D modeltemplate. FIG. 12 illustrates vertices 1202 from a 3D model template andpoints 1204 from a 3D point cloud reconstruction. In some embodiments,for each vertex of the 3D model template, a collection of “several”closest points 1206 in the reconstructed model is determined. In someembodiments, the “several” closest points are based on the 3D modelsresolution. In some embodiments, the “several” closest points may bechosen in a restricted neighborhood 1206 around the normal vector 1214,at the 3D point cloud reconstruction. In some embodiments, centroid 1210of the “several” vertices is computed in 3D. In some embodiments, anamplitude of displacement of the at least some of the points of the 3Dtemplate model and the neighboring points is based on a scalar productof the normalized normal vector (length=1) and the vector linking thevertex to be displaced in the 3D model template and the centroid 1210 inthe 3D point cloud reconstruction, in accordance with the followingformula:

NormalVectorLength×PointToCentroidVectorLength×cos(AngleBetweenThoseTwoVectors)

θ 1212 defines an angle between the normal and the centroid 1210. Thevalue of this angle may be either unconstrained or constrained. In oneexample, the final displacement of a point in vertices 1202 isrepresented by 1208.

In some embodiments, the displacement of the at least some of the pointsof the 3D model template influences its neighbors according to a “bell”function so that the overall displacement of the 3D model templatepreserves the smoothness of the 3D shape. FIG. 13 illustrates the “bell”function at 1302 and scenarios 1304 and 1306 of displacements of thevertices.

FIG. 14 illustrates a first example of a first 3D model template in afirst version 1402, prior to being morphed in accordance with the method600. FIG. 14 also illustrates the first 3D model template in a secondversion 1404, after having been morphed and defining a first 3Drepresentation of the foot captured by a 3D point cloud representation,in accordance with the method 600.

FIG. 15 illustrates a second example of a second 3D model template in afirst version 1502, prior to being morphed in accordance with the method600. FIG. 15 also illustrates the second 3D model template in a secondversion 1504, after having been morphed and defining a second 3Drepresentation of the foot captured by a 3D point cloud representation,in accordance with the method 600.

Turning now to FIG. 16, another example of context for applying themethod 600 is depicted. In this example, the method 600 is executed forthe purpose of body sizing. In such context, the body 1602 illustratedat FIG. 16 may be considered as a collection of rigid objects connectedtogether through geometrical graphs. The method 600 may be applied toeach one of the rigid objects defining the body and then each one of therigid objects may be combined to generate a 3D representation of acomplete body. In some embodiments, a collection of fixed rigid objectmay be considered as a rigid object—e.g. a whole arm from shoulder tofingers can be considered as a rigid object as long as it doesn't move.If it does, each part of the arm will have to be considered separately(arm/forearm/hand/fingers) and reassembled together to form a 3Drepresentation of the body.

Turning now to FIG. 17, a 3D point cloud reconstruction 1702representing a foot and a 3D representation of the foot 1704 generatedfrom the 3D point cloud reconstruction 1702 by applying the method 600are illustrated. As it may be appreciated, the 3D point cloudreconstruction 1702 comprises reconstruction defects on a top portion ofthe foot. In the illustrated example, the top portion of the foot ismissing. Such imperfect 3D point cloud reconstruction of an object mayresult from various causes, for example, but without being limitative,because a portion of an object is not accessible during the 3D pointcloud acquisition step (e.g., a top portion of a foot cannot be accessedduring the scanning, a portion of a screw is located within an assembly,etc.). As another example, defects in the 3D point cloud reconstructionof an object may result from the process of reconstructing the 3D pointcloud reconstruction not having been properly executed Imperfect 3Dpoint cloud reconstructions, such as the 3D point cloud reconstruction1702, are in contrast with “perfect” 3D point cloud reconstructions,such as the 3D point cloud reconstruction 402 illustrated at FIG. 4which is complete, dense and uniform in terms of vertices distribution.

The 3D representation of the foot 1704 reflects the “poor quality” ofthe 3D point cloud reconstruction 1702 in that at least a portion of thefoot (i.e., the toes and the top of the foot in the illustrated example)is not properly reconstructed.

Turning to FIG. 19 and FIG. 20, an embodiment of steps 622-626implementing corrective actions for the processing of an imperfect 3Dpoint cloud reconstruction is exemplified. In some embodiments, thesteps 622-626 may be part of the method 600. In some embodiments, thesteps 622-626 may be executed after step 620.

Step 622 comprises identifying low quality deformation of the 3Drepresentation. The 3D representation may have been generated inaccordance with steps 602-620 based on an imperfect 3D cloudrepresentation, for example the 3D cloud representation 1802. Thelow-quality deformation of the 3D representation may be identified byoperating a quality function. In some embodiments, the quality functionimplements a comparison with a 3D model template. As it may beappreciated, an original 3D model template (i.e., a 3D model templatenot yet subjected to any deformation) and a deformed 3D model template(i.e., the 3D representation generated from the 3D model template) havea similar structure in terms of vertices and faces. It is thus possibleto establish a correspondence between elements of the original 3D modeltemplate and elements of the deformed 3D model template, for example,but without being limitative, a correspondence between a vertex, a faceand/or normal. In some embodiments, elements are each associated with anidentifier (e.g., a number) thereby easing tracking of correspondencesbetween elements.

In some embodiments, the quality function of step 622 may be based ontwo criteria, namely, a distance between two corresponding verticesand/or normal disparities between the original 3D model template and thedeformed 3D model template. It should be understood that the qualityfunction of step 622 may also be based on one of the two criteriawithout departing from the scope of the present technology.

In some embodiments, the distance between two corresponding verticesestablishes 3D distances between the original 3D model template and thedeformed 3D model template. After the 3D point cloud reconstruction andthe 3D model template have been aligned (at steps 606-614), 3D distancesare supposedly minimal. If the 3D point cloud reconstruction reaches acertain quality threshold, 3D local deformations (equally referred to as“displacements”) established at step 618 stays around a certain averageplus standard deviation. If the 3D point cloud reconstruction is below acertain quality threshold, the 3D local deformations are over thestandard deviation. An example of this quality measure is illustrated inFIG. 18, where the 3D model template is given at two different steps ofthe method 600; 1802 is given right after step 614 of the method 600 and1804 after step 620 of the method 600. The distances between thecorresponding vertices of those models whose structure has been kept thesame (including number and indexes of vertices) is given on the 3Ddistance map 1806, as graph 1808 shows the distribution on thosedistances. Since applying method 600 will fit the 3D template model to aminimum distance besides parts that may not be seen, the distributionsgraphs will more or less have a same aspect, i.e. two peaks representingsmall 3D distances and/or good quality (1810) and big 3D distancesand/or bad quality (1812). The vertices to be corrected by steps 622-626of the method 600 may then be easily identified and a quality of the 3Drepresentation may thus be considered as a function of distancesdistribution.

In some embodiments, the normal disparities between the original 3Dmodel template and the deformed 3D model template may be establishedbased on an angle between two corresponding normal of a vertex or aface. A low quality 3D representation (e.g., the 3D representation 1804)may result in a large angle between two corresponding normal. To thecontrary, a good quality 3D representation may result in a small anglebetween two corresponding normals. Again, a 3D quality map may bedisplayed by computing a 3D map of the distribution of 3D normal anglesbetween 3D models 1802 and 1804; the graph will have a similar aspect tograph 1808 and will allow the identification of vertices to becorrected. A quality of the 3D representation may thus be considered asa function of angles distribution.

Additional criteria may also be used in addition or in replacement ofone or both criteria described above without departing from the scope ofthe technology.

Step 624 comprises automatically selecting low quality vertices and/orfaces, considering the definition of quality as explained above. In someembodiments, step 624 may therefore rely on the distances distributionand/or the angles distribution established at step 622 to select the lowquality vertices and/or faces. FIG. 19 illustrates a representation 1902of areas 1910-1914 generated based on step 622 and used for theautomatic selection executed at step 624. Area 1910 represents goodquality parts, meaning that a distances distribution and/or an anglesdistribution between the original 3D model template and the deformed 3Dmodel template is limited. To the contrary, area 1914 represents badquality parts, meaning that a distances distribution and/or an anglesdistribution between the original 3D model template and the deformed 3Dmodel template is above an acceptable threshold. Area 1912 establishes atransition area between good quality parts (i.e., area 1910) and badquality parts (i.e., area 1914) of the 3D representation 1902. In someembodiments, the areas 1910-1914 may be color-coded, area 1910 beingassociated with a blue color, area 1914 being associated with a redcolor and area 1912 being a continuous color gradient from blue to red.

Step 626 comprises correcting bad quality parts based on the 3D modeltemplate. Step 626 may comprise weighting correction of bad qualityparts. In some embodiments, step 626 comprises applying a syntheticdeformation on vertices and/or faces of the bad quality parts identifiedat step 624 (e.g., area 1914) thereby reforming parts of the 3D modeltemplate corresponding to the bad quality parts. In some embodiments,the synthetic deformation is weighted by the quality function of step622 thereby providing a smooth transition between the non-corrected andcorrected parts. An example of steps 622-626 applied to the 3Dreconstruction 1804 is illustrated by the corrected 3D reconstruction1904 of FIG. 19.

As it may appreciated by the person skilled in the art of the presenttechnology, the corrected parts of the 3D reconstruction may notperfectly fit reality but nonetheless allows generating a complete 3Dreconstruction of an object as opposed to a partial 3D reconstructionwith missing parts. It should also be understood that steps 622-626 maynot be systemically applied, in particular if it established at step 620or step 622 that the 3D reconstruction is complete enough and does notrequire corrections.

Notably, the features and examples above are not meant to limit thescope of the present disclosure to a single embodiment, as otherembodiments are possible by way of interchange of some or all of thedescribed or illustrated elements. Moreover, where certain elements ofthe present disclosure can be partially or fully implemented using knowncomponents, only those portions of such known components that arenecessary for an understanding of the present disclosure are described,and detailed descriptions of other portions of such known components areomitted so as not to obscure the disclosure. In the presentspecification, an embodiment showing a singular component should notnecessarily be limited to other embodiments including a plurality of thesame component, and vice-versa, unless explicitly stated otherwiseherein. Moreover, applicants do not intend for any term in thespecification or claims to be ascribed an uncommon or special meaningunless explicitly set forth as such. Further, the present disclosureencompasses present and future known equivalents to the known componentsreferred to herein by way of illustration.

The foregoing description of the specific embodiments so fully revealsthe general nature of the disclosure that others can, by applyingknowledge within the skill of the relevant art(s) (including thecontents of the documents cited and incorporated by reference herein),readily modify and/or adapt for various applications such specificembodiments, without undue experimentation and without departing fromthe general concept of the present disclosure. Such adaptations andmodifications are therefore intended to be within the meaning and rangeof equivalents of the disclosed embodiments, based on the teaching andguidance presented herein. It is to be understood that the phraseologyor terminology herein is for the purpose of description and not oflimitation, such that the terminology or phraseology of the presentspecification is to be interpreted by the skilled artisan in light ofthe teachings and guidance presented herein, in combination with theknowledge of one skilled in the relevant art(s).

While the above-described implementations have been described and shownwith reference to particular steps performed in a particular order, itwill be understood that these steps may be combined, sub-divided, orre-ordered without departing from the teachings of the presenttechnology. The steps may be executed in parallel or in series.Accordingly, the order and grouping of the steps is not a limitation ofthe present technology.

While various embodiments of the present disclosure have been describedabove, it should be understood that they have been presented by way ofexample, and not limitations. It would be apparent to one skilled in therelevant art(s) that various changes in form and detail could be madetherein without departing from the spirit and scope of the disclosure.Thus, the present disclosure should not be limited by any of theabove-described exemplary embodiments, but should be defined only inaccordance with the following claims and their equivalents.

1. A method of generating a three-dimensional (3D) representation of anobject, the method comprising: accessing a 3D point cloud reconstructionof the object; accessing a 3D model template, the 3D model templatedefining a generic version of the object; aligning, in a geometricalspace, the 3D model template with respect to the 3D point cloudreconstruction; adjusting a scale of the 3D model template; and applyinglocal deformations to a surface of the 3D model template.
 2. The methodof claim 1, wherein accessing the 3D point cloud reconstruction of theobject comprises applying a denoising routine.
 3. The method of claim 1,wherein accessing the 3D model template is performed in accordance witha selection routine.
 4. (canceled)
 5. The method of claim 1, wherein thegeometrical space is associated with a cartesian coordinate systemand/or a cylindrical coordinate system and/or a spherical coordinatesystem.
 6. The method of claim 1, wherein aligning the 3D model templatewith respect to the 3D point cloud reconstruction comprises applying arotation and/or a translation.
 7. The method of claim 6, whereinaligning the 3D model template with respect to the 3D point cloudreconstruction comprises: (i) aligning, in translation, a first centerof gravity associated with the 3D model template and a second center ofgravity associated with the 3D point cloud reconstruction; and (ii)aligning, in rotation, the 3D model template with respect to the 3Dpoint cloud reconstruction is based on at least one of a minimumdistance between the 3D model template and the 3D point cloudreconstruction, Principal Components Analysis, and/or 2D projectionsalignments.
 8. The method of claim 7, wherein aligning, in rotation, the3D model template with respect to the 3D point cloud reconstructionbased on Principal Components Analysis comprises identifying 3 main axisof the 3D model template to align the 3 main axis with the axis of the3D point cloud reconstruction.
 9. The method of claim 7, whereinaligning, in rotation, the 3D model template with respect to the 3Dpoint cloud reconstruction based on 2D projections alignments comprisescomputing 2D pictures from the 3D model template in a combination ofdifferent points of view and match the 2D pictures to match the pointsof views.
 10. The method of claim 1, wherein adjusting the scale of the3D model comprises executing an iterative closest point (ICP) algorithm.11. The method of claim 1, wherein adjusting the scale of the 3D modelis performed iteratively and comprises: one of (1) modifying a scale ofthe 3D model template, (2) applying/refining a rotation to the 3D modeltemplate and/or (3) applying/refining a translation to the 3D modeltemplate; calculating a distance between the 3D model template and the3D point cloud reconstruction; and wherein the iterating stops based ona calculated minimum distance.
 12. The method of claim 1, applying localdeformations to the surface of the 3D model template comprises:determining, for at least some of the points of the 3D model template,normal axis extending from each of the at least some of the points ofthe 3D model template; and displacing the at least some of the points ofthe 3D model template along corresponding normal axis of a computedamplitude, the displacing further comprising displacing neighboringpoints located within a radius of the at least some of the points of the3D model template.
 13. The method of claim 12, wherein the computedamplitude is based on a scalar product of a normalized normal vector tothe at least some of the points of the 3D model template and a vectorlinking a point to be displaced in the 3D model template and a centroidin the 3D point cloud reconstruction.
 14. The method of claim 1, furthercomprising: identifying low quality deformations; selecting parts to becorrected; and correcting the parts.
 15. The method of claim 14, whereinlow quality deformations are identified by operating a quality function.16. The method of claim 14, wherein selecting parts to be corrected isbased on distances distribution and/or angles distribution between the3D model template and a deformed 3D model template.
 17. The method ofclaim 1, wherein applying the local deformations to the surface of the3D model template comprises: computing a quality level of the localdeformations; and correcting the local deformations based on thecomputed quality level.
 18. The method of claim 17, wherein thecorrecting the local deformations based on the computed quality levelfurther comprises correcting the local deformations if a computedquality level is below a threshold.
 19. The method of claim 1, furthercomprises generating at least two 3D representations of at least twoobjects and combining the at least two 3D representations. 20.(canceled)
 21. A non-transitory computer-readable medium comprisingcomputer-executable instructions that cause a system to execute themethod according to claim
 1. 22. A system for generating athree-dimensional (3D) representation of an object, the systemcomprising: a processor; a non-transitory computer-readable mediumcomprising instructions, the processor; upon executing the instructions,being configured to cause the processor to: access a 3D point cloudreconstruction of the object; access a 3D model template, the 3D modeltemplate defining a generic version of the object; align, in ageometrical space, the 3D model template with respect to the 3D pointcloud reconstruction; adjust a scale of the 3D model template; and applylocal deformations to a surface of the 3D model template.